It’s in the bag: Stronger supervision for automated face labelling

The objective of this work is automatic labelling of characters in TV video and movies, given weak supervisory information provided by an aligned transcript. We make four contributions: (i) a new strategy for obtaining stronger supervisory information from aligned transcripts; (ii) an explicit model for classifying background characters, based on their face-tracks; and (iii) employing new ConvNet based face features. Each of these contributions delivers a significant boost in performance, and we demonstrate this on standard benchmarks using tracks provided by authors of prior work. Finally, (iv), we also investigate the generalization and strength of the features and classifiers by applying them “in the raw” on new episodes where no supervisory information is used. Overall we achieve a dramatic improvement over the state of the art on both TV series and film datasets, almost saturating performance on some benchmarks.